The invention relates generally to range finder type image sensors, and more particularly to such sensors as may be implemented on a single integrated circuit using CMOS fabrication.
Electronic circuits that provide a measure of distance from the circuit to an object are known in the art, and may be exemplified by system 10 FIG. 1. In the generalized system of FIG. 1, imaging circuitry within system 10 is used to approximate the distance (e.g., Z1, Z2, Z3) to an object 20, the top portion of which is shown more distant from system 10 than is the bottom portion. Typically system 10 will include a light source 30 whose light output is focused by a lens 40 and directed toward the object to be imaged, here object 20. Other prior art systems do not provide an active light source 30 and instead rely upon and indeed require ambient light reflected by the object of interest.
Various fractions of the light from source 30 may be reflected by surface portions of object 20, and is focused by a lens 50. This return light falls upon various detector devices 60, e.g., photodiodes or the like, in an array on an integrated circuit (IC) 70. Devices 60 produce a rendering of the luminosity of an object (e.g., 10) in the scene from which distance data is to be inferred. In some applications devices 60 might be charge coupled devices (CCDs) or arrays of CMOS devices.
CCDs typically are configured in a so-called bucket-brigade whereby light-detected charge by a first CCD is serial-coupled to an adjacent CCD, whose output in turn is coupled to a third CCD, and so on. This bucket-brigade configuration precludes fabricating processing circuitry on the same IC containing the CCD array. Further, CCDs provide a serial readout as opposed to a random readout. For example, if a CCD range finder system were used in a digital zoom lens application, even though most of the relevant data would be provided by a few of the CCDs in the array, it would nonetheless be necessary to readout the entire array to gain access to the relevant data, a time consuming process. In still and some motion photography applications, CCD-based systems might still find utility.
As noted, the upper portion of object 20 is intentionally shown more distant that the lower portion, which is to say distance Z3 greater than Z3 greater than Z1. In an range finder autofocus camera environment, devices 60 approximate average distance from the camera (e.g., from Z=0) to object 10 by examining relative luminosity data obtained from the object. In FIG. 1, the upper portion of object 20 is darker than the lower portion, and presumably is more distant than the lower portion. In a more complicated scene, focal distance to an object or subject standing against a background would be approximated by distinguishing the subject from the background by a change in luminosity. In a range finding binocular application, the field of view is sufficiently small such that all objects in focus are at substantially the same distance. In the various applications, circuits 80, 90, 100 within system 10 would assist in this signal processing. As noted, if IC 70 includes CCDs 60, other processing circuitry such as 80, 90, 100 are formed off-chip.
Unfortunately, reflected luminosity data does not provide a truly accurate rendering of distance because the reflectivity of the object is unknown. Thus, a distant object surface with a shiny surface may reflect as much light (perhaps more) than a closer object surface with a dull finish.
Other focusing systems are known in the art. Infrared (IR) autofocus systems for use in cameras or binoculars produce a single distance value that is an average or a minimum distance to all targets within the field of view. Other camera autofocus systems often require mechanical focusing of the lens onto the subject to determine distance. At best these prior art focus systems can focus a lens onto a single object in a field of view, but cannot simultaneously measure distance for all objects in the field of view.
In general, a reproduction or approximation of original luminosity values in a scene permits the human visual system to understand what objects were present in the scene and to estimate their relative locations stereoscopically. For non-stereoscopic images such as those rendered on an ordinary television screen, the human brain assesses apparent size, distance and shape of objects using past experience. Specialized computer programs can approximate object distance under special conditions.
Stereoscopic images allow a human observer to more accurately judge the distance of an object. However it is challenging for a computer program to judge object distance from a stereoscopic image. Errors are often present, and the required signal processing requires specialized hardware and computation. Stereoscopic images are at best an indirect way to produce a three-dimensional image suitable for direct computer use.
Many applications require directly obtaining a three-dimensional rendering of a scene. But in practice it is difficult to accurately extract distance and velocity data along a viewing axis from luminosity measurements. Nonetheless many application require accurate distance and velocity tracking, for example an assembly line welding robot that must determine the precise distance and speed of the object to be welded. The necessary distance measurements may be erroneous due to varying lighting conditions and other shortcomings noted above. Such applications would benefit from a system that could directly capture three-dimensional imagery.
Although specialized three dimensional imaging systems exist in the nuclear magnetic resonance and scanning laser tomography fields, such systems require substantial equipment expenditures. Further, these systems are obtrusive, and are dedicated to specific tasks, e.g., imaging internal body organs.
In other applications, scanning laser range finding systems raster scan an image by using mirrors to deflect a laser beam in the x-axis and perhaps the y-axis plane. The angle of defection of each mirror is used to determine the coordinate of an image pixel being sampled. Such systems require precision detection of the angle of each mirror to determine which pixel is currently being sampled. Understandably having to provide precision moving mechanical parts add bulk, complexity, and cost to such range finding system. Further, because these systems sample each pixel sequentially, the number of complete image frames that can be sampled per unit time is limited.
Attempts have been made in the prior art to incorporate some logic at each image sensor pixel to process at least some data acquired by the pixel. Such implementations are sometimes referred to as smart pixels. For example, El Gamal et al has attempted to provide special circuitry within pixel to carry-out analog-to-digital conversion of all light sensed by the pixel. Carver Mead and others have configured pixels such that pixels can communicate with to adjacent pixels, in an attempt to directly detect object contours by examining discontinuities in the sensed brightness pattern.
El Gamal, Carver Mead, and other prior art smart pixel approaches to imaging essentially process images based upon overall brightness patterns. But in an image system that seeks to acquire three-dimensional data, the performance requirement for the sensor pixels requires capabilities quite different than what suffices for ordinary brightness acquisition and processing. Further, whereas prior art smart pixel approaches result in brightness-based data that is presented to an image viewable by humans, three-dimensional data should be in a format readily processed and used by digital computer or processor systems.
Thus, there is a need for a new type of smart pixel implantation for use in a direct three-dimensional imaging system, in which parameters of smart pixels in an array could advantageously be controlled dynamically by a processor or computer system that preferably was implemented on the same IC as the detector array. If such an implementation could be provided, various sensor array parameters could be dynamically adjusted to perform optically, as the situation at hand required. Such an IC should further include additional hardware and circuitry to make on-the-fly real-time tradeoffs and adjustments or such parameters to promote processing of detection signals. Preferably such single IC system should be implementable using CMOS fabrication techniques, should require few discrete components, and have no moving components. Optionally, the system should be able to output data from the detectors in a non-sequential or random fashion. Further, there is a need for a system that can be implemented without reliance upon high speed counters and/or shutter mechanisms.
The present invention provides such a system.
Applicants"" parent application and the present invention provide a system that measures distance and velocity data in real time using time-of-flight (TOF) data rather than relying upon luminosity data. Both systems are CMOS-compatible and provide such three-dimensional imaging without requiring moving parts. Both systems may be fabricated on a single IC containing both a two-dimensional array of CMOS-compatible smart TOF pixel detectors that sense photon light energy, sensor array support circuitry, sensor control and processor interface circuitry including a microprocessor and control processor unit. The terms pixel and pixel detector may be used interchangeably herein to refer to a photodiode element (among an array of such elements) that senses incoming photon energy, and circuitry associated with that particular element to enhance a signal output by the pixel responsive to detecting photon energy.
The preferably IC-located microprocessor continuously triggers an energy emitting source (e.g., a laser or LED) whose light output pulses are at least partially reflected by points on the surface of the object to be imaged. The energy emitter may be fabricated off-chip.
Each system includes optical elements including an off-chip filter and lens that focuses incoming energy reflected from the target object to ensure that each pixel detector in the array receives energy only from a single point on the surface of the imaged object. Note that all optical paths from energy emitter to reflecting object surface points to pixel detectors in the array are of equal length. The array of pixel detectors output signals, responsive to sensed photon light energy, and the output signals are processed by analog and digital circuitry that is also on the same IC chip containing the array. The three-dimensional data is acquired by the array of pixel detectors with independent measurements at each pixel. On-chip measured TOF data may be output in random rather than sequential order, and object tracking and other measurements requiring a three-dimensional image are readily made. In the parent application and in the present invention, the overall system can be small, robust and requires relatively few off-chip discrete components, and consumes very low power such that battery operation is possible. IC yield is improved by adopting circuit layout and design techniques such as common centroid, and appropriate device sizing to reduce the effect of process variation upon system operation.
In the parent application, the overall system required extremely high speed counters or a shutter mechanism. Thus, in a first embodiment in the parent application, for each pixel in the two-dimensional array the IC further includes an associated pulse detector, a high speed counter, and access to an on-chip high speed clock. When each light emitted pulse starts, each pixel detector""s counter begins to count clock pulses and accumulates counts until incoming reflected light photons are detected by that pixel. Thus, the accumulated count value in each high speed counter is a direct digital measure of roundtrip TOF from the system to the reflecting object point corresponding to that pixel. On-chip circuitry can use such TOF data to readily simultaneously measure distance and velocity of all points on an object or all objects in a scene.
A second embodiment of the parent application avoided the need for high speed detectors, counters, and clock. In this embodiment, the IC contains a similar two-dimensional array of pixel detectors, and further includes for each pixel a shutter mechanism and a charge integrator. The shutter mechanism turns on or off an output charge path from each pixel detector to the charge integrator, which may be a capacitor. Before a system-emitted light pulse, the microcontroller opens all shutters, which permits each integrator to collect any charge being output by the associated pixel detector. As object-reflected light energy begins to return to the detector array, pixels focused upon closer object surface points will start to detect and output charge, and after a while pixels focused on more distance points will begin to do likewise. Stated differently, integrators associated with such pixels will begin to integrate charge sooner in time. After a time approximating the emitted light pulse width, all shutters are closed (preferably simultaneously), thus terminating further charge accumulations. The accumulated charge magnitude for each pixel provides direct roundtrip TOF data to the object point upon which such pixel is focused. Preferably one set of data is collected with the shutter remaining on for perhaps the period of the emitted light pulse train frequency. Charge gathered during this data set represents point-by-point reflected luminosity for the object surface, which permits correcting for errors caused by more distant but more reflective object surface portion.
After designing the various embodiments of the invention disclosed in the parent application, applicants designed to provide a further design in which fast counters and/or shutters could be eliminated. Propagating fast clock pulses throughout the IC containing the invention without injecting excessive noise components to various circuitry is a challenging task. The alternative embodiment in the parent application avoided high speed counters and relied upon shutter mechanisms. However a new approach was undertaken in the present invention, in implementing a three-dimensional TOF imaging system. As disclosed herein, an analog measurement involving a charge/discharge capacitor signal is used as an alternative to the high speed counters, yet no shutter mechanism is required.
In the present invention, at time To an emitted pulse having pulse width PW is transmitted, and some of the transmitted energy may be reflected back towards the present invention by an object target. Detector photodiodes in the array output a brightness signal B(t) in response to detected reflected energy, and the B(t) signal is integrated, preferably using a capacitor, to produce a ramp-like signal. An elapsed time ET from t0 to when B(t) attains a predetermined threshold value Xref is determined, where the slope of B(t) is B/PW. The maximum value of B is B(t) after integrating over a time equal to the emitted pulse width PW. Time-of-flight TOF represents time from t0 to when a photodiode detector begins to detect energy is determined, where TOF=ETxe2x88x92IT, and where IT is proportional to PW/B. A system processor determines distance from the system to the target object, from TOF and the velocity of light. Each pixel photodiode acquires delay time data and pulse brightness data simultaneously. Thus, IT is calculated using the slope of the ramp-like signal, which slope can be calculated from the perceived brightness. Brightness (B) is the integration value after the entire return pulse has arrived, and since PW of the emitted energy pulse is known, the integration slope is given by B/PW. Thus, given B and ET, IT is subtracted from ET to yield TOF, essentially in real-time, using on-chip signal processing components.